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Summary 

A piloted simulation study has been conducted 
to compare three different input methods of inter- 
acting with a large-screen, multiwindow, “whole- 
flight-deck” (WFD) display for management of 
transport aircraft systems. One input method, the 
“thumball” concept, used a miniature trackball em- 
bedded in a conventional side-arm controller for cur- 
sor movement and numerical entry. The second 
method, the touch screen concept, provided data en- 
try through a capacitive touch screen installed on the 
display surface. The third method, the voice concept, 
used a speech recognition system with input through 
a head- worn microphone. 

No single input concept emerged as the most de- 
sirable method of interacting with the WFD display. 
Subjective results, however, indicated that the voice 
concept was the most preferred method of data en- 
try and had the most potential for future applica- 
tions, provided there are further advancements in 
voice recognition technology. On the other hand, the 
objective results indicated that, overall, the touch 
screen concept was the most effective input method. 
The quantitative results also showed significant dif- 
ferences between the time required to perform spe- 
cific tasks and the input concept employed, with each 
concept providing the best performance relative to 
a specific task. These results suggest that a system 
combining all three input concepts might provide the 
most effective method of interaction. 

Introduction 

The rapidly increasing use of computer systems 
in the cockpit is providing an increase in information 
that can be displayed and managed within the lim- 
ited flight deck environment and, as a consequence, 
more operational capability to the transport aircraft 
crew. Utilization of multiple (four to eight) small- 
screen color cathode ray tubes (CRT’s) as multi- 
mode, multifunction indicators appears to be the 
current solution to consolidate and integrate this in- 
formation. A potential alternative for future appli- 
cation is a large-screen, multiwindow, “whole-flight- 
deck” (WFD) display for the management of aircraft 
systems (refs. 1 to 3). A WFD display accommodates 
most aircraft information on one large CRT (ulti- 
mately to be replaced by one large, flat-panel display) 
in the pilot’s primary head-down field of view. Fig- 
ure 1 shows an example of a WFD display consisting 
of a primary flight display (PFD), a navigation dis- 
play (ND), an automatic guidance and control unit 
(AGCU), a navigation control display unit (NCDU), 
engine displays, an advisory caution and warning sys- 
tem (ACAWS), and a subsystem display area. To 


effectively interact with such a display, pilots must 
be provided with user-friendly man-machine inter- 
face methods. 

A piloted simulation study was conducted to com- 
pare three different input concepts as methods of 
interacting with a large-screen WFD display. The 
“thumball” concept used a miniature trackball em- 
bedded in a conventional side-arm controller; the 
touch screen concept used a capacitive touch screen 
installed on the CRT; and the voice concept utilized a 
voice recognition system installed in a personal com- 
puter and interfaced to the simulation computer. 

A similar experiment was previously conducted 
that also compared three input concepts as methods 
of interacting with a WFD display (ref. 4). The 
present study was done as a follow-on to that work 
to compare one different input concept and improved 
implementations of the others. 

This paper describes each of the three input con- 
cepts and the experimental design in detail. Ob- 
jective data as well as subjective results, garnered 
from pilot questionnaires and discussions, are also 
presented. 

Simulator Characteristics 

Simulator Cockpit 

This study was conducted in the Crew Station 
Systems Research Laboratory using the Advanced 
Display Evaluation Cockpit (fig. 2), a fixed-base, 
part-task research transport simulator outfitted with 
large CRT displays of the type which might be used 
on future transport aircraft (ref. 5, pp. 84 to 88). 
The equipment used consisted of the three study 
input devices, a 19-in. color raster CRT, a side-arm 
controller, and a throttle. Only the pilot side (left 
side) of the simulator cockpit was used for the study. 

Aircraft Model 

The aircraft model utilized for this study was a 
linear representation of the NASA Transport Sys- 
tems Research Vehicle (TSRV), a specially instru- 
mented Boeing 737-100 (ref. 5, pp. 77 to 83). The 
TSRV is a research aircraft that was modified to 
incorporate electronic displays and all-digital flight- 
control computers. The simulated aircraft was con- 
figured for low-speed level flight and was flown in 
the “manual electric” control mode; therefore, there 
was no attitude hold present. The aircraft model, 
implemented in FORTRAN, was hosted on a Digital 
Equipment Corporation VAX-11/780 minicomputer 
at an iteration rate of 15 Hz. 

Graphics Display 

The graphics display created for this experiment 
was a partial representation of a WFD display. A 



WFD display is one in which most of the aircraft 
state information is located on one large CRT; thus 
the need for specialized dedicated instruments is 
eliminated. The large screen was partitioned into 
multiple display formats in various window areas. 
The relative sizes of these areas were under computer 
control to allow selected areas to expand or shrink 
as a function of events. The display for this study 
consisted of a partial primary flight display (PFD), 
engine displays, an advisory caution and warning 
system (ACAWS), a menu for displaying aircraft 
subsystems, and a menu for selecting symbology 
for presentation on a navigation display (fig. 3). 
There was also an area reserved below the PFD for 
displaying prompts to instruct the pilot which task 
to complete next. 

The WFD display was produced on an Adage 
RDS-3000 programmable display generator using the 
Real-Time Animation Package (RAP). RAP is a 
high-level graphics programming language similar to 
the C language. This graphics software ran at an iter- 
ation rate of 2.5 Hz, six times slower than the aircraft 
model, primarily due to the inefficiency of the RAP 
language. As a result, there was a delay between the 
time an input was introduced into the system and 
the time the updated information was displayed on 
the screen. Although the graphics iteration rate was 
slower than that desired for real-time simulation, it 
was anticipated that this rate would have approx- 
imately the same effect on pilot performance with 
each input concept and hence would create little or 
no concept bias in this study. 

In order to obtain the 2.5-Hz graphics iteration 
rate, a navigation display was not included in this 
WFD display. The PFD was also stripped of all 
unnecessary information in order to further reduce 
the time required to render the entire WFD display. 
The information remaining in the partial PFD con- 
sisted of an aircraft symbol, pitch indices, and head- 
ing, roll-angle, airspeed, and altitude numeric indi- 
cators. This information was all that was required to 
perform the basic flying task used in this study. 

The ACAWS was located in a window directly 
beneath the engine displays. This area was event 
driven. When a warning or caution was introduced 
into the system, the engine displays were automat- 
ically reduced in size and the ACAWS area was 
enlarged to list the subsystems containing failures. 
Additionally, the menu used to display the aircraft 
subsystems was reordered, listing the subsystems 
with warnings at the top of the menu in red and 
those with cautions below in yellow (fig. 4). 

The navigation symbology menu of reference 6 
was used in this study. The menu is generally used 
to select various symbols for presentation on the 


navigation display. In this instance, the navigation 
display was not included as part of the WFD display; 
however, the menu was used to provide a typical 
input task for the evaluation of the input concepts. 

Input Concepts 

Input Tasks 

In order to evaluate the usefulness of each input 
concept as a means of interfacing with the WFD 
display, each pilot was asked to perform several tasks. 
There were three basic types of tasks (table I) , which 
are described as follows. 

Task 1 — Display a Subsystem 

This task required changing the active menu (the 
menu that was in use at the time) from the navigation 
symbology menu to the subsystem menu and then 
selecting the indicated subsystem. 

Task 2 — Display the Subsystems Having a 

Warning and a Caution, and Clear Those 

Failures 

This task required the subsystem having the 
warning to be displayed and the failure cleared. The 
same would then be done for the caution failure. 

Task 3 — Change the Reference Altitude by 
1700 Ft 

This task required engaging the numerical entry 
mode on the navigation symbology menu, changing 
the number to the indicated value, and then exiting 
the numerical entry mode. 

Input Devices 

Thumball 

The thumball is a miniature trackball embedded 
in a conventional side-arm controller (fig. 5 and refs. 7 
and 8). The trackball is rotated with the thumb 
to enable interaction with the WFD display. It is 
an input device that provides position information 
in both the horizontal and the vertical direction. 
Output magnitude is directly related to rotation 
speed. 

The thumball concept utilized the thumball com- 
bined with a button and trigger located on the side- 
arm controller. With this system, the pilot was 
able to interact with all aircraft subsystems using 
one hand. Horizontal rotation of the thumball con- 
trolled the active menu designation, indicated by a 
magenta border surrounding the active menu, while 
vertical rotation of the thumball controlled the cur- 
sor within that menu. When the thumball was 
rotated to the right, the subsystem menu became 
active; when moved to the left, the navigation sym- 
bology menu became active. The cursor moved up 
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or down through the active menu when the thumball 
was rotated up or down. A quick rotation caused the 
cursor to move two positions, while a slower rotation 
moved the cursor one position at a time. Both menus 
had a cursor wraparound feature which allowed the 
cursor to move from the topmost menu item to the 
bottommost item and vice versa. 

Task 1, displaying a subsystem, required a maxi- 
mum of five inputs with the thumball system. First, 
the pilot changed the active menu from the navi- 
gation symbology menu to the subsystem menu by 
rotating the thumball to the right. Next, the cur- 
sor was moved three menu locations to the desired 
menu item. The item was then selected by pulling 
the trigger. During task 2, subsystem failure, the 
cursor was initially positioned next to the warning 
item at the start of the task by the ACAWS. As 
a result, cursor movement was not necessary. The 
trigger was pulled to display the subsystem contain- 
ing the warning. The warning was then cleared by 
pressing the right button on the control stick. The 
same steps were conducted for the subsystem con- 
taining the caution (a total of four inputs). The pi- 
lot changed the reference altitude value for task 3 
by moving the cursor three menu locations and then 
engaging the numerical entry mode by pulling the 
trigger. The pilot then changed the reference alti- 
tude by rotating the thumball up to increment and 
down to decrement the value. A slow movement of 
the ball changed the number by 100; a quick move- 
ment changed it by 1000. After the pilot obtained 
the desired value, he exited the enter-number mode 
by pulling the trigger (a minimum of nine inputs). 

Touch Screen 

The touch screen system utilized a 19-in. capaci- 
tive touch screen that required skin contact directly 
with the surface of the screen for activation to occur 
(refs. 9 and 10). This concept operated in an intu- 
itive manner. All that was necessary to activate a 
menu was to touch within the menu itself. To con- 
vey precisely which menu item was being chosen, the 
background color of an item changed when its touch 
zone was being pressed. As long as pressure was ap- 
plied to the screen, nothing was activated. A menu 
item was selected when the finger was released from 
the screen. Performing a selection by this method 
helped prevent inadvertent selection of an undesired 
item. Note that cursor moves were not required for 
the touch screen concept. A direct selection of the 
proper item was possible. 

A subsystem was displayed for task 1 by the pilot 
touching the required item in the subsystem menu 
and exiting from the screen. To complete task 2, the 
subsystem containing the warning was displayed by 


the pilot selecting the appropriate menu item. At 
that time, a “delete” option would be listed in the 
ACAWS area (fig. 6). The warning would then be 
cleared through selection of this option. This pro- 
cess was repeated for the subsystem containing the 
caution. The pilot changed the reference altitude 
value (task 3) by selecting the numerical menu item 
in the navigation symbology menu. As a result, ar- 
rows were displayed at the bottom of the navigation 
symbology menu (fig. 7). Pressing the thicker ar- 
rows changed the number by 1000, while pressing 
the thin arrows resulted in change by 100. The nu- 
merical value was continually altered while pressure 
was placed on an arrow. After the proper value was 
obtained, the enter-number mode was exited by se- 
lecting the numerical menu item again. 

Voice 

The voice concept utilized a speaker-dependent 
template-matching voice recognition system with in- 
put through a head- worn microphone (refs. 11 to 
14). The voice system was operated in a “connected- 
speech” recognition mode that enabled the system 
to recognize phrases embedded within a spoken sen- 
tence. Because of the delay caused by the graphics 
iteration rate, it was necessary for the pilot to pause 
after each phrase; therefore, recognition was more 
comparable to isolated word mode. When a spoken 
phrase was acknowledged by the system, a code as- 
sociated with that phrase was sent to the host com- 
puter. It was necessary to train the system for each 
subject individually for the vocabulary used for the 
simulation. The following list is the 34-phrase vocab- 
ulary set used in this study: 


GRP 

HYDRAULICS 

ZERO 

NAVAIDS 

ELECTRICAL 

ONE 

LOCAL AIRPORTS 

ENVIRONMENT 

TWO 

WAYPOINT DATA 

CHECKLIST 

THREE 

STRAIGHT VECTOR 

SUBSYSTEM 

FOUR 

TREND VECTOR 

DELETE 

FIVE 

RANGE ARC 

ACAWS MENU 

SIX 

REFERENCE ALTITUDE 

SELECT 

SEVEN 

SCAN MODE 

CLEAR 

EIGHT 

SECONDARY ENGINE 

EXIT 

NINER 

FUEL 

ENTER 

HUNDRED 

THOUSAND 


The voice system operated in a manner similar to 
the touch screen system. To select a menu item, the 
subject first had to choose the item by speaking the 
vocabulary phrase for that item and then select the 
item by saying “select.” When the item was chosen, 
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the background color of that item was changed to 
notify the subject that the voice phrase was acknowl- 
edged by the computer. “Select” was used to engage 
the menu item. This two-step method of selection 
helped prevent accidental activation of an undesired 
item. As with the touch screen concept, cursor move- 
ments were not required since direct selection of the 
items was possible. 

Task 1 required the pilot to display a sub- 
system by speaking the vocabulary phrase for the re- 
quested subsystem and then selecting the subsystem 
by saying “select.” Subsystem failures for task 2 were 
cleared by the pilot speaking the phrase for the sub- 
system containing the warning, followed by “select.” 
As the subsystem was being displayed, a delete op- 
tion was listed in the ACAWS area, similar to the 
touch concept (fig. 6). The pilot would clear the 
warning by speaking “delete,” followed by “select.” 
The same would then be done for the subsystem con- 
taining the caution. For example, to clear a warn- 
ing in the electrical subsystem and a caution in the 
environment subsystem, the inputs would be as fol- 
lows: “electrical,” “select,” “delete,” “select,” “envi- 
ronment,” “select,” “delete,” “select.” For task 3, the 
pilot changed the reference altitude value by saying 
“reference altitude,” followed by “select.” A scratch- 
pad area was then displayed below the reference al- 
titude number in the navigation symbology menu 
(fig. 8). The pilot entered the desired value into 
the scratch-pad area by speaking the individual dig- 
its followed by the phrase “thousand” or “hundred.” 
For example, to enter the value 6300, the pilot would 
say “six,” “three,” “hundred.” The reference altitude 
was then equated to the scratch-pad value by the pi- 
lot saying “enter.” The scratch-pad area was utilized 
so the pilot could verify that the proper value was 
entered before activation took place. 

Experiment Description 

Experimental Design 

A full- factorial design with four replications for 
each condition was used for this study. The fac- 
tors of the experiment were pilots, input concepts, 
input tasks, and versions of the various input tasks 
(replicates). 

Each pilot was given four versions of each of the 
three tasks while concentrating on flying the simu- 
lated aircraft (experimental tasks) and also while fo- 
cusing on just the tasks themselves (training tasks). 
This setup resulted in a total of 24 tasks for ev- 
ery input concept. The versions within a task were 
designed to have the same number of cursor moves 
and selections, so each would take approximately the 
same amount of time to perform. The pilot was 


prompted to perform a particular task by a message 
displayed in the center of the screen. 

Objective Performance Measures 

Performance data were collected for each experi- 
mental task. The time to complete each task, from 
the moment the prompt was displayed until the task 
was completed, was recorded. The average altitude 
and airspeed during each experimental task were also 
obtained. 

Experimental Conditions 

Six U.S. Air Force EC-135 pilots and six U.S. 
Air Force C-21 pilots were used in this study. The 
EC-135 pilots had a cumulative total of 16450 flying 
hours, with an 1100-hour minimum and a 4500-hour 
maximum flying time. The C-21 pilots had accumu- 
lated 6825 flying hours, with a 275-hour minimum 
and a 2600-hour maximum flying time. Each pi- 
lot was asked to perform 12 training tasks and then 
12 experimental tasks with each input concept. Ta- 
ble II shows the sequence of input concepts given to 
each pilot. The same 12 tasks were given in each 
instance but in random orders. During the experi- 
mental tasks, predetermined time intervals were ran- 
domly maintained between tasks so the test subject 
would not learn to anticipate when the next trial 
would be given. For the experimental tasks, the pilot 
was asked to maintain straight and level flight dur- 
ing turbulent conditions at an altitude of 1500 ft, an 
airspeed of 130 knots, and a heading of 360°. The 
turbulence was introduced into the model by super- 
imposing noise values, using normally distributed 
random numbers, on the control stick. 

At the start of each session, the pilot was pro- 
vided with training time to become familiar with the 
aircraft simulator. Prior to data collection for each 
input concept, the concept was demonstrated and 
the pilot was allowed to practice with it until he was 
comfortable with the operation. A sample prompt 
for every type of task was also furnished so the pilot 
would know what to expect when the data collec- 
tion began. The training tasks were then given for a 
concept, followed by the experimental tasks for that 
same concept. The training tasks were used only for 
training, and this training allowed the pilot to be- 
come thoroughly familiar with the operation of the 
input concept for the various tasks. 

Data collected during the experimental tasks were 
used in the analysis of the performance measures. 
Input errors were recorded, and those tasks that re- 
sulted in errors were repeated at the end of that ses- 
sion. A subjective questionnaire was given at the 
completion of data collection for each input concept. 
A final subjective questionnaire was also given at the 
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completion of the session to get a comparative opin- 
ion. The questionnaires are shown in appendixes A 
and B. 

Results and Discussion 

Objective Results 

Analysis of Performance Measures 

Table III is a summary of the analyses of variance 
for the three performance measures recorded during 
the experimental tasks, with third-order and higher 
interactions pooled in the error term. Results were 
considered statistically significant at the 95-percent 
confidence level. The following sections discuss some 
of the significant sources of variation for the perfor- 
mance measures. The interactions that are not dis- 
cussed were not considered to be contributing factors 
to the analysis of the data. 

Input concepts. The task completion time and 
average airspeed measures identified significant vari- 
ability among concepts. (See table IV.) The lowest 
mean completion time across all tasks was recorded 
when the touch screen was used. This is under- 
standable since the system operated in such a 
straightforward manner. With this concept, the pilot 
only needed to press the desired item and exit from 
the screen for selection. The voice concept resulted in 
the longest mean completion time. This could be at- 
tributed to the fact that there was a time delay from 
the moment the pilot spoke a phrase to the moment 
the WFD display reacted to the input. Therefore, 
a pause was necessary between each phrase. Also, 
a phrase would have to be repeated occasionally be- 
cause it was not recognized by the voice recognition 
system. The thumball concept resulted in a mean 
completion time slightly slower than that of the touch 
screen concept. Although this method enabled all 
input to be made with one hand from the side-arm 
controller, more time would occasionally be required 
to complete a task because the cursor was sometimes 
driven past the desired menu item and correction was 
necessary. This occurrence was caused by the slow 
graphics iteration rate as well as the rotation rate 
of the ball. These limitations are discussed in more 
detail in the Subjective Results section. 

Since the pilot’s hands were always on the controls 
with the voice or thumball concepts, it was expected 
that the average airspeed would deviate less when 
these systems were used. This expectation held true 
for the voice concept; however, the data did not in- 
dicate this occurrence for the thumball concept. The 
airspeed deviation for the tasks performed with the 
thumball concept may have been caused by inadver- 
tent control stick inputs when rotating the thumball. 


The airspeed variation for the tasks performed with 
the touch screen concept may have also been caused 
by inadvertent control stick inputs when the pilot 
leaned forward to press the screen. 

Tasks. The task factor was statistically significant 
for the performance measures of task completion 
time and average airspeed. Table V shows that 
selecting a subsystem was the quickest task across all 
input concepts, while changing the reference altitude 
was the most time-consuming. These results were 
expected since selecting a subsystem was the simplest 
task to perform and changing the reference altitude 
was the most complicated. 

The mean airspeed during task 2, handling cau- 
tions and warnings, deviated from the means of the 
other two tasks, thus accounting for the significance 
of this interaction term. This occurrence could be 
attributed to the fact that two of the four versions of 
task 2 included an engine failure, which affected air- 
speed by shutting down engine 2. Although the mean 
airspeed during task 2 deviated from the others, it 
was nearer the designated airspeed of 130 knots. 

Interaction of pilot and input concept. The inter- 
action of pilot and input concept for task completion 
time is presented graphically in figure 9 and numeri- 
cally in table VI. In general, all pilots performed best 
with the touch screen concept and worst with the 
voice concept. The first three pilots were an excep- 
tion, with pilots 1 and 3 performing better with the 
thumball concept and pilot 2 performing best with 
the voice concept. Pilot acceptance or preference is 
one explanation as to why performance across pilots 
varied among concepts. Another reason may be the 
physical characteristics of the subjects. For example, 
a pilot with large fingers may not have been able to 
select items easily using the touch screen, or a pilot 
with a temporarily hoarse voice may have had trouble 
getting the voice recognition system to acknowledge 
his commands. 

Interaction of input concept and task. Not un- 
expectedly, the completion time differences between 
the three input concepts varied with the task being 
performed, as indicated by the significance of the 
interaction of input concept and task. Figure 10 
and table VII show that the touch screen concept 
was well suited for tasks which involved on-off oper- 
ations, such as selecting a subsystem (task 1). The 
voice concept was best suited for entering a numerical 
value (task 3). Subsystem failures (task 2) were han- 
dled more effectively with the thumball concept since 
fewer steps were required and the hands remained on 
the controls. 
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Analysis of Input Error Data 

An input error was recorded for a particular 
task if the pilot incorrectly completed the task as 
stated in the prompt. There were 16 errors noted 
for the 432 tasks given during data collection. Ta- 
ble VIII shows the number of errors by task and input 
concept. 

The touch screen concept resulted in the most 
recorded input errors, particularly for entering the 
reference altitude. These errors were caused by 
the pilots exiting the enter-number mode when the 
improper value was entered for the reference altitude. 
The location of the input arrows on the screen along 
with the size of the touch zone for each arrow and the 
spacing between menu items may have contributed 
to the error rate. The pilots occasionally exited 
the enter-number mode prematurely by accidentally 
pressing the reference altitude number menu item. 
Also, the improper numerical value was sometimes 
accepted because the number would be incremented 
or decremented one additional unit after the pilot’s 
finger was removed from an arrow. This was caused 
by the delay in the system. Touch screen errors also 
occurred when an incorrect subsystem was selected 
and when a caution was cleared before a warning. 

The thumball method of entry resulted in pilot 
errors for selecting an incorrect subsystem and ac- 
cepting an improper reference altitude. These errors 
could be attributed to the delay in the system and 
also to the rotation rate of the ball. These problems 
are discussed in more detail in the Subjective Results 
section. 

Note that the voice method of data entry resulted 
in no errors. After becoming familiar with the vo- 
cabulary set, the pilots had no trouble providing 
the proper inputs for the tasks. When the system 
would not recognize an input command repeatedly, 
that task was reiterated but was not recorded as an 
error. 

Any task that resulted in an error was deleted 
from the performance measure set and repeated at 
the end of that session to provide a complete data 
set of correctly performed tasks. 

Subjective Results 

This section discusses the comments obtained 
from the questionnaires shown in the appendixes 
and observations made during data collection. The 
comments are discussed in terms of input concept, 
with general comments following. 

Thumball Concept 

In general, the thumball method of input was the 
least preferred of the three systems. This can be 


largely attributed to the movement of the ball itself. 
Most pilots commented that the ball movement was 
too sensitive in both the vertical and the horizon- 
tal direction. This sensitivity resulted in accidental 
menu changes when the pilot’s thumb moved over 
the ball or when the pilot moved the ball vertically 
at a rapid pace. It was particularly difficult for the 
pilots to distinguish between fine and coarse move- 
ments, where rotating the ball slowly was considered 
a fine input and rotating it quickly was a coarse in- 
put. The pilots suggested that more friction be added 
to the ball so that it would not be so easy to rotate 
or that detents be added so distinct fine or coarse 
positions could be located. Additionally, the thum- 
ball rotation values can be programmed per applica- 
tion and could help alleviate some of these problems. 
A faster graphics iteration rate could also improve 
performance. 

The task of entering numerical data was consid- 
ered fairly difficult and relates directly to the fine- 
coarse movement issue. For this experiment, a fine 
movement changed the altitude value by 100 and a 
coarse movement changed it by 1000. A suggestion 
was made to use a horizontal movement to change the 
numerical value by 1000 and to use a vertical move- 
ment to change the value by 100. Another possibil- 
ity suggested was to change the 100 and 1000 values 
independently, instead of having the values change 
incrementally. Additionally, an occasional input re- 
versal would be given when the pilot changed the 
value. For instance, the ball was sometimes rotated 
downward to increase the number, or vice versa. This 
could be attributed to the fact that pilots associate a 
forward (or upward) movement of the hand controller 
to command a pitch down, or a lowering of the nose 
of the aircraft, and a backward (downward) input to 
command a pitch up. Most pilots felt, however, that 
the direction of movement was appropriate and be- 
lieved errors would not occur with more training with 
the system. 

Pilots reacted positively to using one hand to 
provide all inputs necessary to interface with the 
aircraft systems. The hands were positioned in a 
normal flying arrangement and did not have to be 
removed from the controls during input. It was 
felt that the interaction with subsystem failures was 
particularly easy with this concept. 

The thumball system as a whole was favorably 
noted as having more feedback than the other two 
methods. Something was always moving or changing 
on the screen. This was not so with the touch 
screen and voice systems. However, some pilots 
were concerned that moving a ball embedded in the 
control stick could interfere with flying the aircraft. 
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It was also thought that it may be difficult to operate 
the ball effectively during turbulent conditions. 

Touch Screen Concept 

The touch screen method of entry was well suited 
for tasks which involved on-off operations. It was felt 
that the task of selecting subsystems by touch was 
easy and straightforward. Pilots selected menu items 
by merely touching the desired item and releasing. 

The location of the touch screen was a concern 
for most pilots. They did not feel it was wise to 
have to remove a hand from a control and reach 
forward to press the screen. This could result in 
erroneous control stick movements. It was suggested 
the touch screen be positioned near the throttles so 
it did not require the pilot to reach forward. Pilots 
also found that they were concentrating more on 
the interactive task and were neglecting their flying. 
They thought that in an actual flight environment 
too much time would be spent “inside” the cockpit 
instead of scanning the outside environment. 

The pilots considered the touch screen method 
of entering numerical data to be fairly difficult. The 
location of the input arrows appeared to be the major 
contributing factor. The arrows were positioned at 
the extreme lower left corner of the screen. Pilots 
commented it was awkward to reach diagonally to 
press the arrows and suggested positioning them on 
the right side of the screen. It was also suggested 
that the arrows be displayed higher on the screen or 
that a pop-up menu be used so the arrows could be 
larger in size. Some pilots had difficulty locating the 
touch zone for the arrow. It was suggested that each 
arrow be displayed inside a rectangular area so there 
would be no question where the zone was located. 

Other considerations when using a touch screen 
as an input device relate to the parallax problem. 
If a user’s eye position is not in the same general 
vicinity as the designer’s was when the touch zones 
were defined, it may appear that to select an item 
the user must press below-above or to the left-right 
of the item. One possible solution to the selection 
problem may be to provide a cross hair on the screen 
to show the location where the system recognizes a 
screen press. Also, touch zones should be designed 
large enough to accommodate fingers of all sizes and 
to account for turbulent conditions. Finally, the 
touch screen technology utilized is dependent on the 
application. For instance, Air Force pilots must wear 
gloves during various phases of flight. Therefore, a 
touch screen activated by capacitance could not be 
used because direct contact must be made with the 
skin. 


Voice Concept 

According to pilot discussions, the voice concept 
was, overall, considered the most preferred method of 
input of the three systems evaluated. Operation was 
very straightforward. Menu items were selected by 
verbally stating the desired item and then verbally 
selecting it. Entering numerical data was considered 
easier than with the other two methods. Since this 
implementation did not require physically reaching 
or moving a device, the hands were free for flying the 
aircraft and performing other pertinent tasks. It was 
also noted that it was not necessary to concentrate 
as much on the menus during the tasks. When 
a command was given, it was only necessary to 
glance at the menus to make sure the command was 
recognized. 

A concern with the voice concept was the recog- 
nition capability of the speech recognition system. A 
concentrated effort by the pilot was sometimes neces- 
sary when pronouncing a phrase from the vocabulary 
set. Inflections in the voice pattern for a particular 
phrase had to remain consistent or the phrase may 
not have been recognized by the system. The voice 
recognition system seemed to be rather intolerant to 
variations in the voice pattern. As a result, phrases 
would occasionally have to be repeated to ensure ac- 
knowledgment. Certain phrases were more sensitive 
than others. Some of these sensitive phrases were 
“select,” “delete,” “enter,” “six,” and “hundred.” 

Pilots also commented that the response time for 
command acknowledgment was too slow. There was 
a delay between when a phrase was spoken and when 
a change occurred on the display screen. This was 
due to the difference between the iteration rates of 
the graphics program, the aircraft model, and the 
voice recognition system. The pilots would have 
preferred being able to enter commands in a normal 
speaking voice without having to pause between each 
phrase to ensure recognition. 

As mentioned previously, menu items were se- 
lected with a two-step process. First, the phrase for 
the item itself was spoken; then the item was en- 
gaged by the pilot speaking the phrase “select.” This 
was done in order to avoid inadvertent selection of 
an item. Some pilots felt this two-step process of se- 
lection was redundant and would have preferred en- 
gagement when the item phrase was spoken. When 
developing a voice system, care should be taken to 
utilize the most appropriate selection method for the 
specific application. 

In general, the pilots felt that the voice method 
of input could be useful in a transport environ- 
ment and was the most promising technology, as- 
suming advancements are made in voice recognition 
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technology. Concern with operation in a routine 
cockpit environment was expressed, however. Poten- 
tial problems could arise from radio transmissions, 
noise from decompression, warning bells, or back- 
ground cockpit conversation. Also, in emergency 
conditions, voice inflections would probably change 
dramatically. Voice patterns could also be altered 
when a pilot is ill with a cold or sore throat. Some 
of these concerns could be remedied by turning the 
voice system “on” or “off’ when necessary. Also, if 
voice input is combined with other input methods, 
such as the touch screen or thumball, these systems 
could serve as backups for each other in the event of 
problems. 

General Comments 

Table IX shows that the pilots, in general, pre- 
ferred the touch screen method of entry for selecting 
subsystems, the thumball method for handling sys- 
tem failures, and the voice method for entering the 
reference altitude. These comments conform with the 
performance data. (See fig. 10.) Each of the three 
concepts was considered a good candidate as an in- 
put method for a particular task. This could suggest, 
then, that a combination of input systems would be 
more suitable for providing inputs in a cockpit envi- 
ronment rather than one particular concept. 

Concluding Remarks 

No single input concept emerged as the most de- 
sirable method of interacting with the whole-flight- 
deck display. The objective results of this study in- 
dicated that the touch screen concept was, overall, 


the most effective interface. Operation was intu- 
itively simplistic since the pilots selected items by 
merely touching the desired item and releasing. Sub- 
jectively, favorable reactions to aspects of all three 
input concepts were expressed; however, the voice 
concept was the most preferred method of data en- 
try. The voice concept was also considered to have 
the most potential for future applications, provided 
there are further advancements in voice recognition 
technology. Pilots noted it was desirable to oper- 
ate most aspects of the pilot interface verbally. This 
freed the hands for flying the aircraft and performing 
other pertinent tasks, thus reducing pilot work load. 

The completion time differences between the 
three concepts varied with the task being performed. 
For the present implementations, the touch screen 
concept was well suited for tasks which involved on- 
off operations; entering a numerical value was best 
handled with the voice concept; and subsystem fail- 
ures were handled most effectively with the thumball 
concept. This suggests, therefore, that a combination 
of input concepts might be the most effective method 
of providing system inputs in a cockpit environment 
rather than one particular concept. 

It is recommended that this study be repeated 
in a simulator environment that can provide real- 
time iteration rates for the aircraft model and graphic 
displays. This will determine if the present iteration 
rates had any effect on the results of this study. 

NASA Langley Research Center 
Hampton, VA 23665-5225 
February 23, 1990 
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Appendix A 

Subjective Questionnaire for Individual Input Concepts 

Check one of the following: Pilot # _ 

Thumball 

Touch Screen 

Voice 

For questions 1-4 answer with: 

A - Strongly Disagree With Statement 
B - Mildly Disagree With Statement 
C - Neither Agree Nor Disagree 
D - Mildly Agree With Statement 
E - Strongly Agree With Statement 

__ 1. Entering reference altitude was an easy task. 

2. Making changes (such as selecting subsystems, menus, etc.) was easy. 

3. This system could serve a useful purpose in the air transport environment. 

4. This system could reduce my workload while flying in a “high activity” flight 

environment. 


5. List two or more things (such as characteristics, etc.) you liked about this input system. 


6. List two or more things you did not like about this input system. 


7. Other comments regarding this input system. 



Appendix B 

Final Subjective Questionnaire 

Pilot # 


Please rank order the three systems under consideration for each situation as listed below. (Assume 
only transport operations.) 


A = Thumball 
B = Touch Screen 
C = Voice 


Selecting Handling Entering Ref. 

Subsystems Failures Altitude 


1st 


2nd 


3rd 


Emergency 

Flight 

Conditions 


1st 


2nd 


3rd 


Potential 
Usefulness 
In Transports 


Routine 

Flight 

Conditions 


COMMENTS: 
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Table I. Description of Versions of Input Tasks 


Task 

Version 

Description of task 

1 

1 

Display electrical subsystem 

1 

2 

Display fuel subsystem 

1 

3 

Display secondary engine subsystem 

1 

4 

Display environment subsystem 

2 

1 

Handle engine warning, electrical caution 

2 

2 

Handle fuel warning, engine caution 

2 

3 

Handle electrical warning, environment caution 

2 

4 

Handle environment warning, fuel caution 

3 

1 

Change reference altitude by +1700 ft 

3 

2 

Change reference altitude by —1700 ft 

3 

3 

Change reference altitude by +1700 ft 

3 

4 

Change reference altitude by —1700 ft 


Table II. Sequence of Input Concepts Given to Each Pilot 


Sequence of input concepts 0 for pilot — 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

A 

A 

B 

B 

C 

C 

A 

A 

B 

B 

C 

C 

B 

C 

A 

C 

A 

B 

B 

C 

A 

C 

A 

B 

C 

B 

C 

A 

B 

A 

C 

B 

C 

A 

B 

A 


°Input concepts: A — thumball; B — touch screen; C — voice. 


Table III. Summary of Analyses of Variance for Performance Measures 


Factor 

(a) 

Degrees of 
freedom 

Significance^ of performance 

measures of — 

Time 

Altitude 

Airspeed 

P 

11 

** 

** 

** 

C 

2 

** 

— 

** 

T 

2 

** 

— 

** 

V 

3 

— 

— 

** 

PxC 

22 

** 

— 

** 

P x T 

22 

** 

— 

— 

PxV 

33 

— 

— 

— 

C xT 

4 

** 

— 

— 

CxV 

6 

— 

— 

— 

TxV 

6 

** 

— 

** 

Error 

320 





^Factors are as follows: P — pilot; C — concept; T — task; V — version. 
b Significance shown as follows: 

— not significant at levels considered; 

*significant at 5-pecent level; 

**significant at 1-percent level. 
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Table IV. Means and Standard Deviations for Statistically Significant 
Measures of Input Concept Factor Across all Tasks 


Input 

concept 

Task completion time, sec 

Airspeed, knots 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Thumball 

11.72 

8.85 

133.46 

7.88 

Touch screen 

10.39 

6.40 

132.62 

8.10 

Voice 

13.79 

7.16 

130.69 

9.85 


Table V. Means and Standard Deviations for Statistically Significant 
Measures of Task Factor Across all Input Concepts 


Task 

(a) 

Task completion time, sec 

Airspeed, knots 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

1 

5.59 

2.87 

133.28 

7.70 

2 

12.24 

7.02 

130.16 

10.11 

3 

18.07 

6.55 

133.33 



7.52 


“Tasks are as follows: 

1 — display a subsystem. 

2 — handle a warning and caution. 

3 — change reference altitude. 


Table VI. Means and Standard Deviations for Statistically Significant Measures of 
Interaction of Pilot and Input Concept 


Pilot 


Task completion time, sec, for — 



Thumball concept 

Touch screen concept 

Voice concept 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

1 

10.33 

6.60 

11.67 

7.30 

10.94 

5.26 

2 

17.10 

10.59 

14.83 

9.54 

14.34 

6.46 

3 

8.77 

6.24 

9.24 

6.02 

16.50 

9.13 

4 

12.93 

13.45 

10.88 

6.30 

16.59 

8.17 

5 

12.88 

10.27 

10.18 

6.89 

18.42 

9.56 

6 

11.61 

8.19 

10.31 

4.55 

11.75 

5.94 

7 

13.54 

11.38 

11.04 

7.03 

13.55 

6.52 

8 

10.90 

6.87 

8.59 

4.58 

14.61 

7.94 

9 

9.93 

7.07 

8.48 

5.06 

13.30 

6.79 

10 

10.91 

7.44 

9.59 

4.78 

11.89 

5.21 

11 

9.80 

7.99 

8.67 

7.50 

11.08 

6.00 

12 

11.97 

8.12 

11.15 

5.56 

12.49 

6.05 
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Table VII. Means and Standard Deviations for Statistically Significant Measures of 
Interaction of Input Concept and Task 


Input 

concept 

Task completion time, sec, for — 

Task 1“ 

Task 2“ 

Task 3“ 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Thumball 

7.09 

2.18 

5.88 

2.17 

22.20 

7.76 

Touch screen 

3.92 

1.97 

10.17 

2.49 

17.06 

5.11 

Voice 

5.75 

1.82 

20.66 

4.60 

14.95 

3.98 


“Tasks are as follows: 

1 — display a subsystem. 

2 — handle a warning and caution. 

3 — change reference altitude. 


Table VIII. Number of Input Errors Recorded During Data Collection 


Task 

Number of errors for — 

Thumball 

concept 

Touch screen 
concept 

Voice 

concept 

Display a subsystem 

2 

1 

0 

Handle a warning and caution 

0 

2 

0 

Change reference altitude 

2 

9 

0 


Table IX. Input Concept Preference of Pilots for Various Tasks 



Display a 

Handle a warning 

Change reference 

Preference 

subsystem 

and caution 

altitude 

Most 

Touch screen concept 

Thumball concept 

Voice concept 

Least 

Voice concept 

Voice concept 

Touch screen concept 
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Figure 1. Whole-flight-deck display. 
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Figure 2. Research simulator cockpit. 
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Figure 4. Whole- flight-deck display emphasizing advisory caution and warning system (ACAWS). 
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Figure 7. Whole- flight- deck display showing method of numerical entry for touch concept. 
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Figure 8. Whole-flight- deck display showing method of numerical entry for voice concept. 
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Figure 10. Mean completion times of each input concept for all pilots for each task. 
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